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© System and method for suppressing blocking artifacts In decoded transform coded images. 

© A system and method for image processing is disclosed wherein blocking artifacts are decreased or 
removed in regions of slowly varying intensity, i.e., low activity, in an image reproduction decoded from a 
transform coded image representation. The de-blocking procedure may be applied to JPEG DCT-based 
decoded images, and AC prediction is done selectively only in regions of low activity and only if the AC 
prediction is consistent with known decoded AC coefficient values of lower precision than the predicted values. 
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BACKGROUND OF THE INVENTION 

I. Field of the Invention 

5 The present invention relates to electronic image processing, and more particularly to a system and 
method for suppressing blocking artifacts in regions of slowly varying intensity in an image reproduction 
decoded from a transform coded image representation. 

II. Prior Art 

70 

Various techniques are used in image processing for coding image data to compress and decompress 
image representations. A common technique involves Discrete Cosine Transform (DCT) coding and a 
particular example of image coding using the DCT is found in the JPEG Draft International Standard, 
International Standard DIS 10918-1 (JPEG DIS), and also CCITT Recommendation T.81. As will be 

75 understood, for example from the description in the JPEG DIS, incorporated herein by reference, DCT 
coding involves dividing image data into blocks having associated DC and AC coefficients. In low bit-rate 
DCT compression systems the decoded images usually exhibit blocking artifacts, i.e., stepped edges and 
regions where there are variations in intensity. These artifacts are due to discontinuities between the DCT 
blocks, because of a visible change in the DC level from block to block. At higher bit rates these 

20 discontinuities are removed by low frequency AC coefficients. However, at lower bit rates the low frequency 
AC coefficients are not of sufficient precision to suppress the discontinuities. 

A^ standar d technique for de-blocki ng i s called AC prediction. In AC prediction the low frequency AC 
^ coefficients are predicted from the DC coefficient for a given block and Ith e DC coeffinipnts nf {he eighT 
gurroi innjng nftarftgt npiqhhfir runcks^ Ebis 3x3 array ot block s Is fll UScTtoa quadratic surface and a 

25 thftgr<=>foal D^T computed for the surface. The res ult is a set or predicted AC coefficients based on 
-digQEflQCas between the various DC levels in the 3x3 array. 

This AC prediction technique is well known, and is described in detail in the JPEG DIS in Annex K, 
subclause K.8. It is described there as a decoder option, and the procedure discussed there for applying it 
to a decoder is also disclosed by C.A. GONZALES, ET AL, in C.A. Gonzales, L. Allman, T. McCarthy, P. 

30 Wendt and A.N. Akansu, " DCT Coding for Mntinn Virio n fitnrmgp I isiog Adaptive Arithmetic Codi ng", Signal 
Pmnfl ssj ria: Imaae Communic^ig ps fl f45-:L 5 ri (mn o^ and in C.A. Gonzales and T. McCarthy, "Transform 
Coding using Coefficient Prediction Techniques", U.S Pat.No. r 5,0P1 ,559, issued March 19, 1991, and 
assigned to the same assignee as the present invention. The procedure disclosed in the patent applies AC 
prediction to every block when decoding. As a result, the edge regions in a decompressed reproduced 

35 image may be blurred by the lack of discriminating between blocks in regions of slowly varying intensity 
and blocks in active regions. With this procedure, the AC prediction is thresholded to prevent it from ever 
exceeding a value which would have been quantized as non zero, and therefore transmitted or stored as 
part of the compressed data. The thresholded value is added to the decoded value to do the de-blocking. 
Another approach to solving the block artifacts problem is found in U.S.Pat.No.^ 70T/73& issued to A. 

40 FERRE, ET AL, and entitled "Adaptive Process for the coding and Decoding of a Sequence of Pictures by 
Transformation and Devices for Performing this Process". While this patent teaches "discriminating whether 
each block represents a highly animated or slightly animated scene", the discriminating is for the purpose 
of coding a sequence of images (Col.2, lines 26-28). The Background section mentions the problem of 
"demarcations between the blocks in the uniform zones of the pictures, because the eye is sensitive to 

45 limited brightness differences in uniform zones." (Col 2, lines 7-10). The suggested solution to the problem 
is to transmit or send differences between adjacent images in such slowly varying regions using neighbor- 
hood information within the picture. 

U.S.Pat.No. 4,703,349, issued to J. BERNSTEIN, and entitled "Method and Apparatus for Multi- 
Dimensional Signal Processing Using a Short-Space Fourier Transform," is directed to solving the blocking 

so problem by using over-lapped basis functions. In the Background section, Col. 2, lines 12-46, some of the 
other prior art efforts to reduce the visibility of the "blocking effect" are mentioned. 

A paper by H.C. REEVE III and J.S. LIM, "Reduction of Blocking Effect in Image Coding", ICASSP'83, 
Boston, pp. 1212-1215, describes two methods to reduce blocking: the overlap method and the filtering 
method. It was found that the overlap method increased the bit rate, while the filtering method degraded 

55 edge content. 

C. H. YIM and J.KIM in "A Simple DCT-CVQ based on Two DCT coefficients" Picture Coding 
Symposium '88, pp.8.1 1.1-8.1 1.2, Torino, Italy (Sept. 12-14, 1988) improve upon traditional DCT coding with 
Vector Quantization (VQ) by classifying images based on AC energy properties and edge patterns revealed 



2 



EP 0 585 573 A2 



in the OCT coefficients. This method uses just the two lowest frequency coefficients AC01 and AC10 to 
classify the blocks as homogeneous area and three edges of vertical, horizontal, and diagonal direction. It 
does not use the edge classification to selectively apply AC coefficient prediction. 

K. NEMOTO and T. OMACHI in "Average Separation Method for DCT Coding Using Smooth Interpola- 

5 tion," Picture Coding Symposium *88, pp.8.12.1-8.12.2, Torino, Italy (Sept. 12-14, 1988), describe the 
problem of blockiness distortion occurring in decoded images for low bit rate coding. They solve the 
problem with a technique based on smooth interpolation. The disclosed method does not predict the AC 
coefficients and smooths the active regions as well as the slowly varying regions. 

Another example of AC prediction was disclosed to the Joint Photographic Experts Group (JPEG) in 

70 standards document "Patent Status" by HITACHI, LTD, ISO/IEC JTC1/SC2/WG8 N-432, October 16, 1989. 
It describes the claims in Japanese Patent Application 62-196184, assigned to HITACHI, LTD., as covering 
a method generally involving receiving a DC coefficient, predicting some AC coefficients from a few DC 
coefficients of adjacent blocks, and then doing an inverse discrete cosine transform on the DC coefficient 
and the predicted AC coefficients. Apparently, the method only applies AC prediction to the first stage of 

15 JPEG progressive decoding when only the DC coefficient values have been decoded. No activity measure 
is used to selectively apply AC prediction and no decoded AC coefficients are used to determine if the 
predicted AC values are within a selected range so that a predicted value is used in decoding instead of the 
decoded value. 

U.S.Pat.No. 4,698,689, "Progressive Image Transmission" to KOU-HS TZOU discloses a method of 
20 progressively transmitting and reconstructing an image in which an approximate image is reconstructed 
based upon partial information, and details are added as additional information becomes available. This 
system illustrates how the more coarsely quantized AC coefficient information would be available at low-bit 
rates but it does not predict the finer detail after having transmitted the partial information. 

U.S.Pat.No. 4,774,574, "Adaptive Block Transform Image Coding Method and Apparatus," to S.J. DALY, 
25 ET AL, assigned to Eastman Kodak Company describes edge detection methods as part of an adaptive 
quantization process in which the blocks with high frequency details can be quantized differently from the 
blocks with only low frequency information. These methods do not use the information about frequency to 
selectively apply AC coefficient prediction. 

EPO Patent Application 0283715, "Method of Coding Image Signals" by H. LOHSCHELLER, assigned 
30 to ANT Nachrichtentechnik GmbH, discloses a method that changes the thresholding for the quantization of 
the DCT coefficients as a function of the occurrence of edge structure in the image. The blocks are 
classified block-by-block and there is no use of the edge structure detection to selectively apply AC 
coefficient prediction. 

35 III. Problem to be Solved 

There is therefore a problem in the image processing art with regard to the presence of blocking 
artifacts in regions of slowly varying intensity in image reproductions or reconstructions decoded from 
transform coded image representations and with the availability of satisfactory systems and methods for 
40 removing them. 

Accordingly, it is an object of the present invention to provide a system and method capable of 
achieving improved de-blocking in image reproductions decoded from transform coded image representa- 
tions. 

It is a further object of the present invention to provide a system and method capable of achieving 
45 improved deblocking in decoded transform coded image representations transmitted or stored at low bit 
rates, and, as a particular example, in reproductions or reconstructions of JPEG DCT-based decoded 
images. 

SUMMARY OF THE INVENTION 

50 

The present invention involves a system and method for electronic image processing wherein blocking 
artifacts are decreased or removed in regions of slowly varying intensity, i.e., low activity, in an image 
reproduction decoded from a transform coded image representation. An improved de-blocking procedure is 
disclosed, which may be applied to JPEG DCT-based coded images, and wherein AC prediction is do ne 
55 sejscjbfaly in decoding, i.e., only in regions of low activity and/or only if the prediction for a given fl6 
coefficient is consistent with a Kno wn lower precision value already decoded for that coefficient. 

More particularly, the system of the invention accepts electronic signals indicative of transform coded 
representations of an image for decoding, which signals are specifically indicative of DC and AC coefficient 



3 



EP 0 585 573 A2 



values of compressed blocks of image data representative of the image regions to be decoded. The system 
evaluates whether the region of the image being decoded is varying slowly or rapidly in intensity using a 
suitable technique, i.e., tests for activity to determine whether a region being decoded is of low activity or 
high activity, respectively, and the results of the evaluation are used as follows. 

5 If the results of the testing indicate that the region being tested is varying rapidly in intensity, then no 
blocking correction is applied to the received signals when decoding. If the testing indicates that the region 
is varying slowly, then the blocking correction is applied to the received signals when decoding. 

An alternative or additional technique involves determining whether the AC prediction value of one of 
the blocks of compressed image data received is within a selected range with respect to the received AC 

70 coefficient value, and if the AC prediction value of that one block is within the selected range, the AC 
prediction value is used in the inverse transform of the block. However, if the AC prediction value of that 
one block is not within the selected range, the decoded coefficient value is used in the inverse transform of 
the block. The selected range is preferably the range of original coefficient values that would have been 
quantized to the decoded quantized value during the forward transform coding of the image. 

75 

BRIEF DESCRIPTION OF THE DRAWINGS 

'^Figure 1 illustrates an 8x8 DCT coefficient array showing the DC coefficient and 63 AC coefficients. 

Figure 2 is an illustration of a typical DCT block and the eight nearest neighbors used for AC pred iction 
2<T within tha t block. 

" Figure 3 shows an AC coefficient patter n used for z onal classification . 

Figure 4 illustrates the relationship between the levels of the dequant iz ed predicted AC c oefficient and 
the dequantized decoded AC coefficient AC/, and adjacent dequantized levels, AC/+1 and AC/-"7 



25 DESCRIPTION OF THE PREFERRED EMBODIMENTS 



The present invention consists of a system and method that involve a new way of applying the standard 
technique for de-blocking called "AC prediction", which standard technique is describ ed in detail, for 
example, in the above-cited JPEG DIS Annex K, subclause K.8. In AC prediction the ^jow freque ncy AC 
30 coefficients of t h ^j^2gn fc in an ima g° block aa^Eredfcfed from the DC coefficiejiL of that block aiTcrtrTe" 
DC co eff idelTts^oTtneeight surrounding nearest neighbo r blocks as shown in Figure j) Thi s 3x3 array of „ 
hg?^i<^ittft(JtoIa guadratic ^urface and a theoretical DCT is computed for the surface. Jhe_rgfi"fr a 
of jjredicted AC coefficients based oru di fferences between the various DC^ levels in the 3x3 array, which 
co efficients are used in decoding the transform coded image representatio ns. 
35 r More particularly, the system of the invention accepts electronic signals, i.e., receives transmitted or 
stored signals, indicati ve of transform cod ed representations of an image for decoding, which signals are 
specifically indicative ^f DC and AC coefflcTe Ytt-yalues of compressed blocks of image data representative 
of the image regions to be decoded. The system evaluates whether the region of the image being decoded 
is varying slowly or rapidly in intensity by using at least one of the following techniques: 
40 spatial edge detection; 

frequency edge detection; 
zonal classification; and 
threshold classification. 

Spatial edge detection can be done by any of a number of well known techniques such as gradient 
45 operators, compass operators, zero crossings and stochastic gradients. The Sobel operator is one example 
of a spatial edge detection operator that may be found suitable, the details of which are described, for 
instance, in "Digital Image Processing," R.C. GONZALES and P. WINTZ, Ad di son-Wesley Publishing Co., 
Reading, Massachusetts, 1987, 2nd Edition. Another reference containing details on this and the other 
techniques is "Fundamentals of Digital Image Processing", A.K. JAIN, Prentice Hall, Englewood Cliffs, NJ, 
50 1989, chapter 9. 

Frequency edge detection techniques are applied in the transform domain. Since a direct relationship 
exists between geometrical features in a DCT block and the distribution of AC coefficients for that block, 
frequency edge detection can be done by looking at the features of the transformed block. 

Zonal classification is a well known way to measure activity in an image. For example, a good model of 
a region of an image with slowly varying intensity can be achieved by allowing only the two lowest C 
ft-ftr|uftn.fi Y AC coefficients. ACQ1 and AC10, to be non zero as shown in Fig. 3. yJ 
In threshold classifica tion th Q ^Hty nf oap h block is determined fry means of thresholds for each AC 
< ^ coefficient. 
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These activity classification techniques make use of some or all of the following parameters: 

DC coefficients of the block being tested and at least one neighboring block; 

DC coefficients of blocks which are neighbors of the block being tested; 

one or more AC coefficients of the block being tested; and 
5 one or more AC coefficients of blocks which are neighbors of the block being tested. 

The AC prediction described in detail in the JPEG DIS Annex K, subclause K.8, is described as a 
decoder option, and the procedure discussed for applying it tn a decoder, ^a s in the above-cited C.A. 
GONZALES ET AL references, suggests tha t the^&Cp red ictionjje thresholdelTT p prevent it from ever 
exceeding a value which would have beer guytized^TTTOfC Sr^and therefore transmitted or stored as 
io part of the compressed data. The thresholcted value is added to the decoded value to do the de-blocking 
when decoding. 

In the present invention this procedure is preferably modified in two important ways which significantly 
improve the effectiveness of the de-blocking and which may be used alone or together. 

1. An activity measure is applie d to each block in the 3x3 array used for AC prediction, and only if a ll 

/5 blocks in the 3x3 array are of now activj& (as opposed to hi gh activity^ is AC p redi ction allowed in 
de1co"drng_the-b lock in question. T f^-waSun tr jaUba^Uivily o f alf of the blocks in the 3x3 array should be 
^considered is that when a block exhib*its(high ac tivity >the DC value for that block is very likely to be 
influenced by the local variations in that bloclctJonsequently, Mi spredictions of AC values in neigh boring 
blocks based on the DC v alue of that active block are likely to be wrong . ~— — * 

20 "2. If the 3x3 array is entirely low activity, the AC prediction is computed from the DC values to a 
precision which is higher than the precision of the decoded AC coefficients. The predicted value is then 
quantized to the same precision as the decoded value. If the decoded value is equal to the quantized 
predicted value, the correction is. accepted. If the values are inconsistent, the decoded value is used 
without modification. 

25 Alternatively, the predicted dequantized AC coefficient can be compared to the dequantized decoded 
values as shown in Figure 4. The predicted value is accepted in place of the decoded value if the predicted 
value falls within a range which, after quantization and dequantization, matches the decoded value. 
A pseudo-code implementation of this scheme is as follows: 

30 For each DCT block: 

Measure activity for each block in the 3x3 block 

array 

If any block is (high activity ) v ^ 

use only / ^eccfttedrg gquantized AC vaToW^ 
else 

For each (low frequ ency DCT^ c oef f icient jj ^Block^ 

40 Compute the Cpredicted values 

Quantize to same precision as decoded value 
If quantized predicted value = decoded value 

45 use predicted AC value 

else 

use decoded dequantized AC value 
endif 

endif 



35 



50 



This combines both the activity measure and the consistency of the AC prediction with the decoded AC 
value. 

55 The activity measure depends on the AC coefficient information known at the time the de-blocking is 
carried out. If only DC information is known, a standard operator such as the Sobel edge detection operator, 
noted above, can be used to determine activity. With suitable thresholding it will determine when AC 
Prediction can be applied. 
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75 



20 



However, if the AC coefficient information is known, it h as been found that a simple measure of act ivity 
can be derived JroTTHhe position" of tha finH-nf-fr lo ck (EOB ). The EOB provides a^^ asuTe^oTtheliig h 
'ffeqU ^ncyj/ariations in the block, and it has been determined t hat a hlonk Rhoij[n^hQ^om<:iy Qred active if 
any^ot the au coerncieats beyon d thejowest frequency hor izontal and vertical -terms^arV]%nz efo] Th i o -Slt 

The de-Sleeking scheme of the invention has been implemented and found to be very effective. For 
example .^results qbtained. using the standard "Lena" image sig nificantly remove blocl^g_artifacjsJ.n,smooth 
regions. The de-blocking is effective even in the smooth regions wherelhe decoded low frequency AC 
•Coefficients are not zero. Furthermore, the activity m easure prevent s wrong AC predictions in regions 
containing edges an d high activity . 

— the invention can oe embodied in a system consisting of hardware or software or a combination of the 
two. For example, a_ DCT chjp_may--be~~usedHn- combination with -a-eomputer— and~software doing 
decompression and AC prediction. Also, a digital signal processor may be programmecLtOHnclude the 
invention as part of its JPEGjdecpdjng function. Suitable programmable video compression processor chips 
for this purpose may be.pbtained, forexarr^ 2445 Mission 

College Boulevard, Santa Clara, CA 95054. 

While the invention has been described with reference to a preferred embodiment thereof, that is, 
specifically as applied to still image JPEG alg orithmsTirwiiTbe understood by those ski lled. in the art that 
various changes in form. ,and„de tails may be made to adapt to other applications, suc h as moving picture 
sequences, without dep3_rting_from~the^cope~o^ For example, the invention has been 

illustrated using the 8x8 DCJTjjlock size in keeping with^ JPEQH5u^ equally well to other block sizes 
and other Jransforms. 



Claims 

25 

1. A method for removing blocking artifacts in regions of slowly varying intensity in an electronic image 
decoded from a transform coded representation of an image, comprising the steps of: 

receiving signals indicative of DC and AC coefficient values of transformed blocks of image data 
representative of the image regions; 
30 evaluating whether the region of the image being decoded is varying slowly or rapidly in intensity, and 
producing signals indicative of the results of said evaluation; 

when said evaluation signals indicate said region is varying rapidly, applying no blocking correction to 
the received signals; and 

when said evaluation signals indicate said region is varying slowly, applying a blocking correction to 
35 said received signals. 

2. A method as in claim 1, wherein said evaluating step comprises using at least one of the following: 

spatial edge detection; 
frequency edge detection; 
40 zonal classification; and 

threshold classification. 



3. A method as in claim 1 or 2, wherein said evaluating step comprises using at least one of the following 
parameters: 

45 DC coefficients of the block being tested and at least one neighboring block; 

DC coefficients of blocks which are neighbors of the block being tested; 
one or more AC coefficients of the block being tested; and 

one or more AC coefficients of blocks which are neighbors of the block being tested. 

so 4. A method for removing blocking artifacts in regions of slowly varying intensity in an electronic image 
decoded from a transform coded representation of an original image, comprising the steps of: 
receiving signals indicative of DC and AC coefficient values of transformed blocks of image data 
representative of the image regions; 

determining whether the AC prediction value of one of said blocks of transformed image data is within a 
55 selected range with respect to the received AC coefficient value; and 

if said AC prediction value of said one block is within said selected range, using said AC prediction 
value in decoding said block; and 

if said AC prediction value of said one block is not within said selected range, using said received AC 
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coefficient value in decoding said block. 

5. A method as in claim 4, wherein said selected range comprises the range of original coefficient values 
that would have been quantized to the decoded quantized value during the forward transform coding of 

5 said original image block. 

6. A method as in claim 4 or 5, wherein said determining step comprises using at least one of the 
following: 

spatial edge detection; 
w frequency edge detection; 

zonal classification; and 
threshold classification. 

7. A method for removing blocking artifacts in regions of slowly varying intensity in an electronic image 
75 decoded from a transform coded representation of an image, comprising the steps of: 

receiving signals indicative of DC and AC coefficient values of transformed blocks of image data 
representative of the image regions; 

evaluating whether the region of the image being decoded is varying slowly or rapidly in intensity, and 
producing signals indicative of the results of said evaluation; 
20 when said evaluation signals indicate said region is varying rapidly, applying no blocking correction to 
the received signals; and 

when said evaluation signals indicate said region is varying slowly, applying a blocking correction to 
said received signals, and wherein said blocking correction comprises the steps of: 
determining whether the AC prediction value of one of said blocks of compressed image data is within 
25 a selected range with respect to the received AC coefficient value, and producing a signal indicative of 
the results of said determination; 

when said signal indicative of the results of said determination indicates said AC prediction value of 
said one block is within said selected range, using said AC prediction value in decoding said block; and 
when said signal indicative of the results of said determination indicates said AC prediction value of 
30 said one block is not within said selected range, using said received AC coefficient value in decoding 
said block, 

wherein said selected range preferably comprises the range of original coefficient values that would 
have been quantized to the decoded quantized value during the forward transform coding of said 
original image block and/or 
35 wherein said evaluating step preferably comprises using at least one of the following: 

spatial edge detection; 

frequency edge detection; 

zonal classification; and 

threshold classification. 

40 

8. A system for removing block artifacts in regions of slowly varying intensity in an electronic image 
decoded from a transform coded representation of an image, comprising: 

means for receiving signals indicative of DC and AC coefficient values of compressed blocks of image 
data representative of the image regions to be decoded; 
45 means, responsive to signals from said receiving means, for evaluating whether the region of the image 
being decoded is varying slowly or rapidly in intensity, and producing a signal indicative of the results 
of said evaluating; 

means for applying a blocking correction to said received signals when said signal indicative of said 
evaluating results indicates said region is varying slowly; and 
so means for avoiding a blocking correction to said received signals when said signal indicative of said 
evaluating results indicates said region is varying rapidly, 

wherein said evaluating means preferably comprises means for testing using at least one of the 
following: 

spatial edge detection; 
55 frequency edge detection; 

zonal classification; and 

threshold classification. 
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9. A system for removing blocking artifacts in regions of slowly varying intensity in an electronic image 
decoded from a transform coded representation of an original image, comprising: 

means for receiving signals indicative of DC and AC coefficient values of transformed blocks of image 
data representative of the image regions; 

means, responsive to signals from said receiving means, for determining whether the AC prediction 
value of one of said blocks of transformed image data is within a selected range with respect to the 
received AC coefficient value and producing a signal indicative of the results of said determination: 
means for using said AC prediction value in decoding said block when said results signal indicates that 
said AC prediction value of said one block is within said selected range; and 

means for using said received AC coefficient value in decoding said block when said results signal 
indicates that said AC prediction value of said one block is not within said selected range, 
wherein said means for determining said selected range preferably comprises means for determining 
the range of original coefficient values that would have been quantized to the decoded quantized value 
during the forward transform coding of said original image block. 

10. A method as in claim 9, wherein said means for determining comprises means for testing using at least 
one of the following: 

spatial edge detection; 
frequency edge detection; 
zonal classification; and 
threshold classification. 
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